<html>
<head><meta charset="utf-8"><title>half of test suite fails since time immemorial · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html">half of test suite fails since time immemorial</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="133893169"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893169" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893169">(Sep 13 2018 at 15:27)</a>:</h4>
<p><span class="user-mention" data-user-id="123586">@nagisa</span> wrote this during today's weekly meeting:</p>
<blockquote>
<p>for me half of the test suite consistently fails all the time since time immemorial</p>
</blockquote>



<a name="133893187"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893187" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893187">(Sep 13 2018 at 15:27)</a>:</h4>
<p>Now, I too often see test failures (especially ones related to LTO)</p>



<a name="133893235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893235">(Sep 13 2018 at 15:28)</a>:</h4>
<p>but the scenario alluded to above seems particularly bad</p>



<a name="133893245"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893245" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893245">(Sep 13 2018 at 15:28)</a>:</h4>
<p>and I don't just mean in terms of personal frustration and/or UX</p>



<a name="133893262"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893262" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893262">(Sep 13 2018 at 15:28)</a>:</h4>
<p>I mean in terms of bugs that are not caught early (and are thus left to travis to catch), simply because the tests aren't ... portable?</p>



<a name="133893289"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893289" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893289">(Sep 13 2018 at 15:29)</a>:</h4>
<p>yup, I have a constant grievance with our test suite</p>



<a name="133893293"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893293" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893293">(Sep 13 2018 at 15:29)</a>:</h4>
<p><span class="user-mention" data-user-id="123586">@nagisa</span> can you provide more info on this? What kinds of tests? And/or hypothetheses about why, e.g. is it some particularlity of your platform?</p>



<a name="133893311"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893311" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893311">(Sep 13 2018 at 15:29)</a>:</h4>
<p><span class="user-mention" data-user-id="116013">@qmx</span> anyone who's experiencing similar issues should feel free to chime in</p>



<a name="133893316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893316">(Sep 13 2018 at 15:29)</a>:</h4>
<p>specially with gdb tests and things that need root permissions and are not obvious at first sight</p>



<a name="133893325"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893325" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893325">(Sep 13 2018 at 15:29)</a>:</h4>
<p>ah okay, that's a good qualifier</p>



<a name="133893389"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893389" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893389">(Sep 13 2018 at 15:30)</a>:</h4>
<p>so <code>run-make-fulldeps</code> is the one that fails more frequently</p>



<a name="133893395"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893395" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893395">(Sep 13 2018 at 15:30)</a>:</h4>
<p>I myself tend to do my regular testing with <code>x.py test src/test/{compile-fail,ui,run-pass,mir-opt}</code></p>



<a name="133893401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893401">(Sep 13 2018 at 15:30)</a>:</h4>
<p>followed by anytime I dare to run the tests in incremental mode</p>



<a name="133893405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893405">(Sep 13 2018 at 15:30)</a>:</h4>
<p>so now I can understand why I'm not seeing the failures described by others</p>



<a name="133893416"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893416" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> davidtwco <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893416">(Sep 13 2018 at 15:31)</a>:</h4>
<p>Generally, I only ever run compile-fail, run-pass or ui locally - unless something fails in another suite on Travis. Every few months I get a failure in UI or something like that locally that doesn't happen on Travis and I ignore. Eventually it goes away.</p>



<a name="133893417"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133893417" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133893417">(Sep 13 2018 at 15:31)</a>:</h4>
<p>then it's random crashes, and I just go and do a full clean build to get the tests to pass</p>



<a name="133894343"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133894343" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> varkor <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133894343">(Sep 13 2018 at 15:45)</a>:</h4>
<p>the main offenders for me are <a href="https://github.com/rust-lang/rust/issues/52528" target="_blank" title="https://github.com/rust-lang/rust/issues/52528">https://github.com/rust-lang/rust/issues/52528</a></p>



<a name="133894386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133894386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> varkor <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133894386">(Sep 13 2018 at 15:46)</a>:</h4>
<p>if it gets past those, it's usually fine iirc</p>



<a name="133895891"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133895891" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133895891">(Sep 13 2018 at 16:11)</a>:</h4>
<blockquote>
<p>specially with gdb tests and things that need root permissions and are not obvious at first sight</p>
</blockquote>
<p>I do have problems with gdb tests too, I don't know how to make them work</p>



<a name="133895946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133895946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133895946">(Sep 13 2018 at 16:12)</a>:</h4>
<p>I use incremental a lot, though; <span class="user-mention" data-user-id="116013">@qmx</span> what problems does incremental cause for you?</p>



<a name="133895969"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133895969" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133895969">(Sep 13 2018 at 16:12)</a>:</h4>
<p>just that sometimes 50% of the test suite fails at random</p>



<a name="133895982"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133895982" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133895982">(Sep 13 2018 at 16:13)</a>:</h4>
<p>then I know it's incremental and something got busted</p>



<a name="133895995"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133895995" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133895995">(Sep 13 2018 at 16:13)</a>:</h4>
<p>never looked into figuring out why</p>



<a name="133895999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133895999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133895999">(Sep 13 2018 at 16:13)</a>:</h4>
<p>huh, ok</p>



<a name="133900707"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133900707" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133900707">(Sep 13 2018 at 17:39)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> my failures are usually related to system setup</p>



<a name="133900716"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133900716" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133900716">(Sep 13 2018 at 17:39)</a>:</h4>
<p>such as debuginfo tests that always fail due to some gdb incompatibility</p>



<a name="133900783"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133900783" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133900783">(Sep 13 2018 at 17:40)</a>:</h4>
<p>there never was a time when I could just run <code>make test</code> and now <code>x.py test</code> and have it just work.</p>



<a name="133900901"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133900901" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133900901">(Sep 13 2018 at 17:42)</a>:</h4>
<p>I also have a few run-pass tests that consistently fail, but I do not remember which ones. I suspect them to be related to system setup as well</p>



<a name="133900932"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133900932" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133900932">(Sep 13 2018 at 17:43)</a>:</h4>
<p>I don’t bother investigating them much as my work usually does not need the whole test suite to pass for me to be confident in my work (I usually just run <code>x.py test src/test/suite --test-args=certain-test(s)</code>.</p>



<a name="133961990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/133961990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#133961990">(Sep 14 2018 at 15:49)</a>:</h4>
<p>BTW <span class="user-mention" data-user-id="121053">@varkor</span> and <span class="user-mention" data-user-id="124288">@Oli</span> , I don't know how well you two manage your Github notifications (i know I simply cannot keep up with mine), but if you get a chance to respond to <a href="https://github.com/rust-lang/rust/issues/52528#issuecomment-421397048" target="_blank" title="https://github.com/rust-lang/rust/issues/52528#issuecomment-421397048">https://github.com/rust-lang/rust/issues/52528#issuecomment-421397048</a> that would be great</p>



<a name="134117055"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/half%20of%20test%20suite%20fails%20since%20time%20immemorial/near/134117055" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/half.20of.20test.20suite.20fails.20since.20time.20immemorial.html#134117055">(Sep 17 2018 at 18:07)</a>:</h4>
<p>never ran tests without optimisation either.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>